Display system in a vehicle

ABSTRACT

A display system in a vehicle, including: one or more displays; at least one sensor determining a user&#39;s line of sight; a controller determining an active zone and a non-active zone of the one or more displays based on the user&#39;s line of sight. The one or more displays are configured to operate the active zone at an enhanced level as compared to the non-active zone. There is further provided a controller configured to rank content displayed on one or more displays in a vehicle according to an attention score determined by at least one sensor configured to measure gaze of a user based on the user&#39;s line of sight; and a display system including the controller. One of the displays is selected to display the highest ranked content. There is also provided a method of operating one or more displays in a vehicle using the disclosed display systems.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is the U.S. National Phase Application of PCTInternational Application No. PCT/EP2018/076898, filed Oct. 3, 2018,which claims priority to Great Britain Patent Application No. 1716146.4,filed Oct. 4, 2017, the contents of such applications being incorporatedby reference herein.

FIELD OF INVENTION

This invention relates to display systems adapted for vehicles.

BACKGROUND OF INVENTION

To enhance driving experience, the number and size of displays in thevehicle and the information provided on these displays are everincreasing. However, this trend could lead to increased driverdistraction and result in safety concerns, particularly when moredisplays and information are within the driver's line of sight while thedriver is operating the vehicle. The various operating displays may alsocause driver fatigue. Driver fatigue may also be caused by having toadapt to different lighting conditions of the various displays orbetween ambient lighting and interior cabin lighting.

Further, as a result of the increased amount of information to bedisplayed, each piece of information may need to be reduced in size tofit within the displays. Drivers may thus require a longer time to readand identify the information required from each display among all theinformation provided on all of the displays. This also contributes toincreased driver distraction.

There is therefore a need to provide vehicle displays that overcome orat least ameliorate one or more of the disadvantages discussed above.

SUMMARY OF THE INVENTION

An aspect of the invention is a display system in a vehicle to addressthe problems discussed above. Particularly, an aspect of this inventionis a display system in a vehicle that provides reduced driverdistraction. A further aspect of this invention is a display system in avehicle which displays information desired by the driver or a passengerand in a manner that is efficiently received by the driver or thepassenger.

To accomplish this, there is provided, in a first aspect, a displaysystem in a vehicle, the system comprising: one or more displays; atleast one sensor configured to determine a user's line of sight; acontroller configured to determine an active zone and a non-active zoneof the one or more displays based on the user's line of sight, whereinthe one or more displays are configured to operate the active zone at anenhanced level as compared to the non-active zone.

A display in a vehicle may include any physical screen used toelectronically display the output of a computer or processor. Thedisplay may optionally provide an interface for interaction with thedriver or user. Thus, the display may provide information to and receiveinformation from the driver or user. Typical displays found in a vehicleinclude instrument clusters, head up displays, any secondary displaysuch as those in the centre stack, and any other displays for vehiclessuch as digital rear view mirror, digital side view mirror, windshield,sun visor, and faceplate. The display may be configured to display asingle-screen display or a multi-screen display. That is, in amulti-screen display, the single physical screen can include multipledisplays that are managed as separate logical displays. Thus, differentcontent can be displayed on a multi-screen display although they arepart of the same physical screen. Accordingly, each display or physicaldisplay of the vehicle may comprise one logical display or multiplelogical displays.

Content may include those typically displayed in vehicles, such as gaugereadings, speedometer readings, fuel level, engine or gear settings suchas vehicle driving mode, sports mode, economy mode, manual geartransmission mode, automatic gear transmission mode, electronic maps,navigation instructions, direction of travel, route destination, time ofarrival, traffic information, speed limits, time, weather, climatecontrol settings, radio settings, details of media content, videos,button selection symbols, and content synced from other devices such ascalendar events, instant messages, emails, notifications, social mediacontent, and calls. As may be appreciated, content may be pulled fromsources external to the disclosed display system, e.g. navigationinstructions may be pulled from a navigation unit or calendar events maybe pulled from connected devices.

A vehicle finding utility with the present disclosure is notparticularly limited as long as the vehicle comprises one or moredisplays. Exemplary vehicles include cars, trucks, vans, boats, andplanes.

The disclosed system comprises one or more sensors configured todetermine a user's line of sight. The sensor assists to track where theuser is looking at or focusing on. The sensor may further assist todistinguish facial gestures or expressions, such as squinting of eyes,frowning, raised eyebrow(s), or other gestures as disclosed herein.Sensors suitable for the present purpose include those that trackmovement of the user's eyes or head or body, such as interior camera,head movement sensor or body movement sensor.

Interior cameras or optical sensors gather raw optical data, using forexample colour CMOS sensor, time of flight sensor, etc., to track theuser's eyes. The tracking and prediction of user's line of sight, alsoknown as eye-gaze detection, are done with known methods such as patternrecognition, corneal reflection points, dark and bright pupil effect orcombinations thereof. Where the user's eyes are partially or fullyobscured, the head or body movement sensor can assist to determine thedirection of user's view and estimate the line of sight. Head movementsensors are capable of sensing the head's lateral movements and rotationmovements (e.g. roll, yaw and pitch). Body movement sensors are capableof sensing movement of a subject's body in a plurality of directions.Therefore, such head or body movement sensors are capable of detectinggestures from the user. In some instances, such optical sensors or heador body movement sensors may be individual devices on its own. In otherinstances, the sensors may be integrated as one device; for example, thehead or body movement sensors may be embedded or integrated into theinterior camera.

The sensor(s) may constantly monitor user's eyes, head and/or bodymovement and predict the user's line of sight. The user's line of sightis therefore used to determine which display or displays and/or whichcontent is required or desired by the user. The display(s) and/orcontent are then grouped into areas by the controller. That is, thecontroller takes in input from the sensor(s) and maps the input intoareas or zones within the group of displays and/or within a singledisplay. The area in which the user is gazing at or focusing on istermed the “active zone”. Consequently, all other display areas will beconsidered the “non-active zone”.

The present disclosure provides assistance in enhancing the area ofinterest to make it easier for the driver or user to identify and/orread information in this area of interest. Accordingly, the timerequired by the driver for such task may be reduced and the driver mayreturn attention to the main task of operating the vehicle, therebyreducing driver distraction. Advantageously, operation of the display(s)in the vehicle may become natural to the driver, instead of being adistraction to the driver. The instantaneous and temporary operation ofthe active and non-active zones also allow the system to provide avariable power mode, thereby extending the lifetime of the system.

The term “user”, as used throughout the present disclosure, refers toany operators of the display system, including the driver as well as anyoccupants or passengers of the vehicle.

Hence, terms such as “a”, “an” and “the” are not intended to refer toonly a singular entity, such as a single user, but include the generalclass of which a specific example may be used for illustration.

The display system may be configured appropriately according to designintent in the event that multiple users are using the display system atthe same time. The display system may be configured to allow multipleusers to use the system at the same time. For example, when more thanone user's line of sight is detected on one display, each line of sightdetermines one active zone which can be operated as disclosed herein.Say when a driver is looking at a gauge reading displayed on anintegrated infotainment and instrument cluster single display, and afront passenger is looking at media content displayed on the sameintegrated display, the gauge reading and the media content each becomean active zone. Alternatively, the display system may be configured toallow certain displays to be used only by certain users. For example,the instrument cluster, head up display, and side and rear view mirrordisplays may be predefined for use only by a user in the driver's seat,while the video display may be predefined for use only by a user in thepassengers' seats. In another example, only certain displays in thevehicle may be configured as part of the disclosed display system.

Operation of the active and non-active zones may comprise modificationof an operating parameter of the one or more displays selected from thegroup consisting of: illumination; movement of any object on thedisplay; magnification; and combinations thereof. The active andnon-active zones may be operated to permit driver attention on thetraffic ahead. Prior art displays may be distracting especially when thedisplays all have high brightness, high colour contrast, constantlychanging content, or may otherwise be sensitive to the eyes.Advantageously, the present disclosure provides that only when certaininformation or content is required by the user will the active zonecontaining such content increase display readability and/or enhanceinformation presentation.

Modification of the operating parameter of the display(s) may beexecuted by a controller of the display. That is, the controller of thedisplay system may transmit a telegram message via a bus network orother appropriate network to an appropriate controller of the relevantdisplay, e.g. the graphics processor or the LED driver, to execute themodification of the operating parameter, e.g. adjust brightness, gammavalues, grey-scale values, zooming in, zooming out, or other parametersas disclosed herein.

In an example, the active zone may be operated at full illumination,e.g. at full colour and/or brightness. Where prior art vehicle displaysmay all be operated at the same colour and/or brightness, the disclosedsystem may only operate the active zone at full colour and/orbrightness. The distraction from lesser important display(s) and/orcontent in the non-active zone at any point in time may be reduced orremoved. Furthermore, in this example, power may be saved as only asubset of the displays, i.e. the active zone, is operated at enhancedillumination. Consequently, thermal dissipation due to power dissipationmay be reduced.

Conversely, the non-active zone may be operated at lower illumination,e.g. lower brightness, lower contrast, and/or lower grey-scale value, ascompared to the active zone. That is, where prior art vehicle displaysmay all be operated at the same colour and/or brightness, the disclosedsystem may dim or darken or grey off the non-active zone, whilemaintaining higher brightness, higher contrast, and/or coloured graphicsin the active zone. The disclosed system may even switch off thenon-active zone, while maintaining the brightness, contrast and/orcoloured graphics in the active zone. For example, the brightness of theactive zone may be at least 2, 3, 4 or 5 times that of the non-activezone. In day time, brightness of the active zone may be 600 cd/m², whilebrightness of the non-active zone may be 100 cd/m²; at night, brightnessof the active zone may be 100 cd/m², while brightness of the non-activezone may be 10 cd/m². Similarly, the grey-scale value of the active zonemay be at least 2, 3, 4 or 5 times that of the non-active zone. In daytime, ratio of the grey-scale value of the active zone to the non-activezone may be 6:1; at night, ratio of the grey-scale value of the activezone to the non-active zone may be 10:1.

Lowering the brightness of the non-active zone may be done in anysuitable way, e.g. by decreasing the backlight brightness of thedisplay(s) in the non-active zone. Lowering the contrast may be done inany suitable way, e.g. by adjusting the gamma value to tune both thedark and bright areas in all three primary colours to be darker, whileincreasing the contrast in the active zone may mean that the gamma valueis adjusted so that the details in the dark areas or shadows areclearer. Greying off the non-active zone may be done in any suitableway, e.g. where the highest grey-scale values make an image appearwhite, the grey-scale value in the non-active zone may be decreased togrey or darken out the non-active zone.

Thus, the operating parameter(s) of the active zone may be modified suchthat the display(s) in the active zone have higher perceived brightnessand/or contrast to the user as compared to the non-active zone. Whileprior art displays may only be capable of adjusting brightness acrossall displays, the disclosed system is advantageously capable ofadjusting individual illumination parameters for each display. Despitethe ability to customize each display of the disclosed system, thesystem as a whole remains easy to use.

In another example, movement of any object on the display may besuspended in the non-active zone. Examples of moving or dynamic objectson a display include speedometer readings, navigation guidance,scrolling of media details such as song title, song album and artist,and videos. Any moving or dynamic object may be frozen when suchinformation is not required, i.e. in the non-active zone. When the userdesires once again to view the dynamic object, the sensor(s) will detectthe user's line of sight and the display displaying the suspended objectwill become the active zone. Thus, the suspended object may once againmove as designed to show, e.g., the continuously updated speed, thecurrent song, etc. Advantageously, driver distraction from movement ondisplay(s) which are not required at the time may be reduced as thechanging graphics can be a source of distraction.

The manufacturer may include overrides or exceptions into the controllerto cater for situations when driver or user attention to the particularcontent or display is needed. Exemplary overrides include criticalwarnings, alerts for driver to take over a semi or fully autonomousdriving mode, and maintaining full operation of the digital side viewmirror when other vehicles are close by. Under such overrides, thecorresponding display will not be affected by the zoning and subsequentoperation of the active and non-active zones but will maintain fulloperation, e.g. brightness, contrast and display of dynamic objects.

As mentioned above, the disclosed system may comprise one display ormultiple displays. Where the system comprises one display, the activezone and non-active zone are within the one display. Where the systemcomprises more than one display, the active zone may comprise onedisplay or more than one display (but less than the total number ofdisplays). For example, the driver may be looking generally at thecentre stack which houses a navigation display between air-conditioningvents and a secondary display above the navigation display. Thus, thecontroller may determine the navigation display and the secondarydisplay to be the active zone, while other displays such as the head-updisplay and the instrument cluster may be determined as the non-activezone. Therefore in this example, the active zone may comprise more thanone display (termed herein as a global active zone). If the driverfocusses further on a part of the navigation display, e.g. a destinationpoint, the active zone is now determined to be that part of thenavigation display, while other parts of the map in the navigationdisplay are determined as the non-active zone. In this example, theother parts of the map may be darkened out, while the destination pointmay be retained in colour and may further be magnified or enlarged foreasy viewing. Thus in this example, the active and non-active zones arewithin the one display (termed herein as a local active zone).

Accordingly, where the active zone is within one display, the activezone may be magnified. The disclosed sensor(s) in the system areadvantageously capable of identifying and differentiating a local activezone from within a global active zone. Particular information that theuser wishes to see in a local active zone may therefore be suitablyenhanced or magnified or zoomed in for quicker identification by theuser. When the user stops looking at the local active zone, i.e. theline of sight is discontinued, the magnified local active zone will thenshrink back to its normal size. In some instances, magnification of theactive zone may be activated upon a command from the user, therebyallowing the user to retain control of the presentation of informationon the display.

The shape and size of the active zone, whether a local or global activezone, are not particularly limited and may be adjustable as determinedby display intention or design intention. For example, a global activezone may include one full display; one full display and half of a seconddisplay; two full displays, etc. In another example, a local active zonewithin a navigation display may include a particular point of interestas well as its surrounding neighbourhood. Alternatively, a local activezone within a media display may include just the song title or just thetime progress bar. The separation between the active and non-activezones may or may not be a fixed separation line and may be adjustable asdetermined by display intention or design intention. The enlargedgraphics of the active zone may overlay onto part of the non-activezone. Furthermore, the granularity of a zone may be adjustable asdetermined by display intention or design intention. For example, amagnified local active zone of a navigation display may have highergranularity, i.e. finer images, than a magnified local active zone of amedia display. Generally, the zones are defined so as to providecomfortable usage to the user.

The display(s) may be connected to an audio output, such as a speaker,to allow objects on the display to be accompanied by sound. For example,navigation instructions may be accompanied by audio instructions so thatthe driver can see as well as hear the turn-by-turn instructions. Wherethe one or more displays are connected to an audio output, and where anobject displayed in the active zone is accompanied by audio, operationof the active zone may comprise at least one of: an increased volume ofthe audio output, or a repeat of the audio output. A combination mayinvolve repeating the audio output by paraphrasing the instruction at anincreased volume to advantageously achieve greater clarity ofinstruction. The line of sight data may determine such operation of theaudio output. For example, upon hearing navigation instructions that thedriver is unclear of, the driver may look at the navigation displaywhich becomes the active zone and the audio output may be modifiedaccordingly. Alternatively or additionally, operation of the audiooutput may be activated upon a command from the user.

The line of sight data may determine the active and non-active zones andthe system may automatically operate the active zone at an enhancedlevel as compared to the non-active zone. Alternatively or additionally,operation of the active and non-active zones may be activated upon acommand from the user, such as magnification of the active zone oroperation of the audio output of the active zone. Thus, the user mayretain at least some control of the operation of the zones.

In instances where operation of the active zone is activated upon acommand, the disclosed system may further comprise an input means toreceive the command from the user. The command received by the inputmeans may be transmitted to the disclosed controller and subsequently toan appropriate controller to execute the command. The input means maycomprise an audio sensor such as a microphone to receive a voice commandfrom the user. Such audio sensor gathers acoustic feedback from the userand/or the surrounding and provides the feedback to the controller. Theinput means may comprise a selection command or haptic input on one ofthe displays for selection by the user. The input means may comprise amovement sensor, such as the interior camera, head movement sensorand/or body movement sensor disclosed herein, to receive a gesturecommand from the user. Alternatively, the input means may be acombination of the mentioned input means. Thus, in this implementation,if the input means detects an affirmative response from the user, thesystem proceeds to operate the active zone appropriately.

For example, a gesture of the driver's head moving forward in thedriving direction may be programmed as a command to activate themagnification of the active zone. In this example, the interior cameraand/or the head movement sensor may determine the driver's line of sightand the controller may determine a global active zone based on the lineof sight data, resulting in diminishing the operation of, e.g. darkeningout, the display(s) in the non-active zone. Thereafter, the headmovement sensor may detect the gesture of the driver's head movingforward, while maintaining line of sight on the content, and thecontroller may determine a local active zone based on the gesturecommand, resulting in the enlargement of graphics in the local activezone. The local non-active zone may remain in colour, i.e. not darkenedout; however the magnification of the local non-active zone may remainunchanged. Operation of the global non-active zone and the localnon-active zone may be programmed as per display or design intent.Alternatively, a spoken command, such as “zoom in”, may be programmed asa voice command to activate the magnification of the active zone, whilemaintaining line of sight on the content. Yet another alternative wouldbe to utilize a display to display the question “zoom in?” and selectionbuttons “yes” and “no” to activate the magnification of the active zone.

In another example, a gesture of the driver demonstrating confusion,e.g. eye rolling, head tilting, and/or frowning, may be programmed as acommand to repeat an audio turn-by-turn route instruction of anavigation display in the active zone. In this example, the interiorcamera may determine the driver's line of sight on the navigationdisplay, resulting in diminishing the operation of all other displayswhich are in the non-active zone. Thereafter, the head movement sensorand/or the interior camera may detect the facial gesture from the driverand the controller may activate a repeat of the route instructionthrough the audio output.

In some instances, the system may automatically operate the active zoneat an enhanced level as compared to the non-active zone based on theline of sight data and the driver may provide a command to further tunethe operation of the active zone according to his preference.

The commands, e.g. gesture command, voice command and/or selectioncommand, may be predetermined by the manufacturer and pre-programmedinto the disclosed display system. Alternatively or additionally, thedisclosed display system may detect or learn common gestures and/orwords from the user and update the collection of commands in the programof the display system. The system may thus advantageously be scalable sothat the user can build further commands into the system according togestures and/or words he is familiar with.

Thus, the disclosed display system may further comprise an artificialintelligence (AI) software connected to the controller so that thesystem may learn from data obtained from the at least one sensor and/orthe input means and refine the operation or display output of the activeand non-active zones. Advantageously, the AI software may providemachine learning capabilities to the disclosed system to improverobustness of the decisions made by the system. The AI software mayutilize information from the cloud or a database to infer how otherusers operate the system. For example, users with similar demographicsmay be concerned with certain content more than other content. Thus, theAI software may tap on this knowledge to improve the system.Additionally, positive or negative reinforcements from gesture and/orvoice commands from the user may be detected and used by the AIsoftware. For example, the driver or user saying “oh yes” or nodding hishead may be programmed to be interpreted as a positive reinforcement ofthe controller, e.g., magnifying a certain part of the display deemed tobe the active zone. Thus, when similar line of sight or other data aredetected by the disclosed system in the future, the controller willissue the same command to magnify the same part of the display.Conversely, the driver or user saying “no no no” or shaking his head maybe programmed to be interpreted as a negative reinforcement so that thedisplay output may be reassessed, the appropriate operating parametermodified and the operation of the active and non-active zones refined.Thus, the user's intent of which display(s) should be selected as theactive zone based on the line of sight sensor data and the user's intentof which operating parameter(s) of the active and non-active zonesshould be modified and how they should be modified may be continuouslyrefined by the AI software. The AI software may therefore be configuredto compare display output with user intent.

The controller may be programmed to prioritize display intent or designintent in order to maintain driving safety and to minimize confusion.

The controller disclosed above may be further configured to rank contentdisplayed on each display according to an attention score determined bya gaze measurement of the user based on the user's line of sight. Asmentioned above, the disclosed sensor(s) assist to track where thedriver or user is looking at or focusing on, i.e., the user's gaze. Eachtime the driver or user looks at a content, the attention score of thatcontent increases. The attention score may be determined by the numberof gazes on a particular content, duration of each gaze on thatparticular content, or a combination thereof. The exact calculation ofthe attention score may be determined according to design intent. Thehighest ranked content represents the content that the user is mostconcerned about. In some instances, the highest ranked content may becopied to or moved to, e.g., a display in the active zone or apreselected display such as the head up display, so that the user mayhave easy access to the most desired information in the most visibledisplay preferred by the user. Alternatively, the display selected todisplay the highest ranked content may be selected by the manufactureror the user, e.g. the driver. In some instances, the controller maydecide if it is necessary to display the highest ranked content on theadditional display. For example, the controller may decide that thehighest ranked content may be presented in other ways for easy access bythe user, e.g. by magnifying the highest ranked content. In anotherexample, the controller may decide if it is not appropriate to displaythe highest ranked content on an additional display, e.g. if the highestranked content is a video. Advantageously, the disclosed display systemdisplays information desired by the user in a manner that is efficientlyreceived by the user. Further advantageously, driver distraction may bereduced and the driver may maintain focus on operating the vehicle whilereferring to the desired information displayed on a convenient display.Even further advantageously, content may be conveniently and dynamicallyprovided to the user based on his needs.

In some instances, the ranking of content may be undertaken for a periodof time and the highest ranked content at the end of that time may bedisplayed in a selected display or otherwise presented for easy accessby the user.

In some instances, the display of the highest ranked content on aselected display or otherwise presented for easy access by the user maybe activated upon a command from the user, thereby allowing the user toretain control of the presentation of information on the display. Thecommand may be one as disclosed herein. In instances where the systemfurther comprises an AI software, if a command is repeated frequentlyenough, the controller may automatically display the highest rankedcontent on the display previously selected. The ranking of the contentmay be reset upon a command from the user or after a period of time. Theuser may determine via a command or otherwise whether the highest rankedcontent should continue being copied to the active zone or thepreselected display.

In some instances, the controller may be configured to determine whetherthe user's gaze or user's line of sight is genuine before the active andnon-active zones are determined and before counting of the attentionscore is started. Genuineness of gaze may be determined by any suitablemeans. For example, once a threshold timing of, e.g., 0.1 seconds haspassed, the controller may then proceed to determine the active andnon-active zones and/or start counting the attention score of content.Gazes below the threshold timing may not be considered by the controllerand may be considered a false trigger, thus no changes to the display(s)will occur.

There is therefore provided, in a second aspect, a controller configuredto rank content displayed on one or more displays in a vehicle accordingto an attention score determined by at least one sensor configured tomeasure gaze of a user based on the user's line of sight.

As mentioned above, the attention score may be determined by at leastone of: number of gazes on a content or duration of each gaze on thecontent. The content ranked may be one as disclosed herein. In someinstances, the content ranked is selected from the group consisting of:driving mode, calendar event, route destination, traffic information,weather, time, and a combination thereof.

The display system may only include the controller of the second aspect.That is, in a third aspect, there is provided a display system in avehicle, the system comprising: one or more displays; at least onesensor configured to determine a user's line of sight; the controller ofthe second aspect, wherein one of the displays is selected to displaythe highest ranked content. As mentioned above, since the highest rankedcontent represents the content that the user is most concerned about, itis advantageous that the highest ranked content is presented in a mannerthat is easily accessible by the user. The manner of presentation of thehighest ranked content may be one as disclosed herein. For example, thehighest ranked content may be copied to or moved to a display desired bythe user, for example the display most visible to the driver.

There is provided, in a fourth aspect, a method of operating one or moredisplays in a vehicle using the system of the third aspect, the methodcomprising: determining, by the at least one sensor, a user's line ofsight; determining, by the controller, a ranking of content displayed oneach display according to an attention score determined by gaze of theuser based on the user's line of sight; selecting one of the displays todisplay the highest ranked content; displaying the highest rankedcontent on the selected display.

In some instances, selection of the display to display the highestranked content may be undertaken by the manufacturer or the user via anysuitable input means, such as those disclosed herein. In some instances,selection of the display to display the highest ranked content may beundertaken by the controller. In such instances, the controller may beconfigured to determine which display is most suitable for display ofthe highest ranked content, for example the display most visible to thedriver or user or a display in an active zone. Accordingly, thecontroller may be one as disclosed herein.

The display(s) and the sensor(s) may be selected from those disclosedherein. The ranking of content and the attention score may be determinedin a manner as disclosed herein.

There is provided, in a fifth aspect, a method of operating one or moredisplays in a vehicle using the system of the first aspect, the methodcomprising: determining, by at least one sensor, a user's line of sight;determining, by a controller, an active zone and a non-active zone ofthe one or more displays based on the user's line of sight; operatingthe active zone of the one or more displays at an enhanced level ascompared to the non-active zone.

The display(s) and the sensor(s) may be selected from those disclosedherein.

The step of operating the active zone at an enhanced level as comparedto the non-active zone may be undertaken in a manner as disclosedherein. For example, operating the active and non-active zones maycomprise modifying an operating parameter of the one or more displaysselected from the group consisting of: illumination; movement of anyobject on the display; magnification; and combinations thereof. In aspecific example, operating the active and non-active zones may compriseat least one of: decreasing brightness of the non-active zone ascompared to the active zone; decreasing contrast of the non-active zoneas compared to the active zone; decreasing a grey-scale value of thenon-active zone as compared to the active zone; or suspending movementof any object on the display of the non-active zone. In another example,where the active zone is within one display, operating the active zonemay comprise magnifying the active zone. In yet another example, wherean object displayed in the active zone is accompanied by audio,operating the active zone may comprise at least one of: increasing thevolume of the audio output; or repeating the audio output.

As mentioned herein, the step of operating the active and non-activezones may be activated upon a command from the user. Thus, the methodmay further comprise receiving, by an input means, a command from theuser. The input means may be one as disclosed herein, for example anaudio sensor to receive a voice command from the user, a selectioncommand on a display, a movement sensor to receive a gesture commandfrom the user, or combinations thereof.

An example of an operating parameter that may be activated upon acommand from the user includes the audio output, e.g. sound from aspeaker. In this example, when a command from the user is received, thesteps of increasing the volume of the audio output of the active zoneand/or repeating the audio output of the active zone are activated.

The disclosed methods may further comprise using data obtained from theat least one sensor and/or the input means in an artificial intelligencesoftware to compare display output with user intent. Examples of dataobtained from the sensor(s) and/or the input means to refine theoperation or display output of the display(s) and/or the active andnon-active zones are as disclosed herein.

The description of aspects of this invention is provided for the purposeof explaining the principles of aspects of the invention and itspractical application, thereby enabling person skilled in the art tounderstand aspects of the invention for various exemplary embodimentsand with various modifications as are suited to the particular usecontemplated. The detailed description is not intended to be exhaustiveor to limit aspects of the invention to the precise embodimentsdisclosed. Modifications and equivalents will be apparent topractitioners skilled in this art and are encompassed within the spiritand scope of the appended claims.

For example, the disclosed display system may be used in a virtualreality headset, gaming console, mobile device, multiple screen setup orhome assistant, or may be used in theatre.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 shows an illustration of the architecture of a display system 100according to a first embodiment of this invention.

FIG. 2 shows an illustration of the architecture of a display system 200according to a second embodiment of this invention.

FIG. 3A shows an illustration of a prior art display system 200′.

FIG. 3B shows an illustration of display system 200 according to thesecond embodiment of this invention, wherein display 208-1 belongs tothe active zone and displays 208-2 to 208-8 belong to the non-activezone.

FIG. 3C shows an illustration of an instrument cluster display 208-2according to the second embodiment of this invention.

FIG. 3D shows an illustration of the instrument cluster display 208-2according to the second embodiment of this invention, wherein theodometer in display 208-2 is magnified.

FIG. 4 shows an illustration of the architecture of a display system 300according to a third embodiment of this invention.

In the figures, like numerals denote like parts.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, exemplary embodiments of the present invention will bedescribed in detail with reference to the accompanying drawings.

FIG. 1 shows an illustration of the architecture of a display system 100according to a first embodiment of this invention. The display system100 comprises p displays 108; an interior camera 102 and a head movementsensor 103, both configured to determine a driver's line of sight; acontroller 106 configured to determine an active zone and a non-activezone of the p displays 108 based on the driver's line of sight, whereinthe p displays 108 are configured to operate the active zone at anenhanced level as compared to the non-active zone.

The interior camera 102 constantly monitors the eye movement of driver110 to predict the driver's line of sight. Where the interior camera 102is unable to detect the eyes of driver 110 to accurately predict thedriver's line of sight, the head movement sensor 103 monitors the headmovement of driver 110 to determine the direction of view of the driverand predict the driver's line of sight. The driver's line of sight isused to determine which display or displays the driver 110 is gazing at.The gaze position of the driver 110 represents the display area that thedriver 110 is most concerned about. The controller 106 maps the gazeposition into areas or zones within the group of displays 108. The areain which the driver 110 is gazing at is termed the “active zone”.

Here, displays 1 and 2 are determined as the active zone, denoted as“a”, and displays 3 to p are determined as the non-active zone, denotedas “n”. Thus, system 100 comprises more than one display, i.e. pdisplays, and the active zone comprises more than one display, i.e.display 1 and display 2.

Operation of the active zone at an enhanced level compared to thenon-active zone may comprise modification of an operating parameter ofthe displays 108 selected from the group consisting of: illumination;movement of any object on the display; magnification; and combinationsthereof. For example, illumination of displays 3 to p in the non-activezone may be modified to have at least one of lower brightness, lowercontrast, and/or lower grey-scale value, as compared to displays 1 and 2in the active zone. Another possible modification is to suspend themovement of any object on displays 3 to p in the non-active zone.

As to how the p displays 108 are operated, the method of operating pdisplays 108 in a vehicle using the system 100 comprises: determining,by interior camera 102 and head movement sensor 103, the line of sightof driver 110; determining, by controller 106, an active zone and anon-active zone of the p displays 108 based on the driver's line ofsight; operating the active zone of the p displays 108 at an enhancedlevel as compared to the non-active zone.

As mentioned above, operating the active zone at an enhanced levelcompared to the non-active zone may comprise modifying an operatingparameter of the one or more displays selected from the group consistingof: illumination; movement of any object on the display; magnification;and combinations thereof. For example, operating the active andnon-active zones may comprise at least one of: decreasing brightness ofthe non-active zone as compared to the active zone; decreasing contrastof the non-active zone as compared to the active zone; decreasing agrey-scale value of the non-active zone as compared to the active zone;suspending movement of any object on the display of the non-active zone.

Say the modification in this case is to suspend the movement of anyobject on displays 3 to p and to grey off displays 3 to p. Thecontroller 106 determines and processes the active and non-active zonesignal, and transmits a telegram to the controller (not shown) ofdisplays 108. When the telegram is received by the controller ofdisplays 108, the controller of displays 108 identifies that displays 1and 2 belong to the active zone and displays 3 to p belong to thenon-active zone. The controller of displays 108 then proceeds to suspendthe movement of any object in the non-active zone and decrease thegrey-scale value of the non-active zone. As compared to the non-activezone, the active zone, i.e. displays 1 and 2, is adjusted or maintainedwith colour at higher brightness and higher contrast. Thus, displays 1and 2 are operated at an enhanced level as compared to displays 3 to pbecause displays 1 and 2 have higher perceived brightness and contrastto the driver as compared to displays 3 to p.

FIG. 2 shows an illustration of the architecture of a display system 200according to a second embodiment of this invention. The display system200 comprises p displays 208; an interior camera 202, a head movementsensor 203 and a body movement sensor 204, all configured to determine adriver's line of sight; a controller 206 configured to determine anactive zone and a non-active zone of the p displays 208 based on thedriver's line of sight, wherein the p displays 208 are configured tooperate the active zone at an enhanced level as compared to thenon-active zone. The controller 206 further comprises an artificialintelligence software configured to use data obtained from interiorcamera 202, head movement sensor 203 and body movement sensor 204 tocompare display output with driver intent.

The interior camera 202 constantly monitors the eye movement of driver210 to predict the driver's line of sight. Where the interior camera 202is unable to detect the eyes of driver 210 to accurately predict thedriver's line of sight, the head movement sensor 203 and/or bodymovement sensor 204 monitors the head and body movement, respectively,of driver 210 to determine the direction of view of the driver andpredict the driver's line of sight. The driver's line of sight is usedto determine which display or displays the driver 210 is gazing at. Thegaze position of the driver 210 represents the display area that thedriver 210 is most concerned about. The controller 206 maps the gazeposition into areas or zones within the group of displays 208. The areain which the driver 210 is gazing at is termed the “active zone”.

Here, display 1 is determined as the active zone, denoted as “a”, anddisplays 2 to p are determined as the non-active zone, denoted as “n”.Thus, system 200 comprises more than one display, i.e. p displays, andthe active zone comprises one display, i.e. display 1.

Operation of the active zone at an enhanced level compared to thenon-active zone may comprise modification of an operating parameter ofthe displays 208 selected from the group consisting of: illumination;movement of any object on the display; magnification; and combinationsthereof. For example, illumination of displays 2 to p in the non-activezone may be modified to have at least one of lower brightness, lowercontrast, and/or lower grey-scale value, as compared to display 1 in theactive zone.

Say the modification in this case is to dim displays 2 to p. After thecontroller 206 determines the gaze of driver 210 to be genuine, thecontroller 206 determines and processes the active and non-active zonesignal, and transmits a telegram to the controller (not shown) ofdisplays 208. When the telegram is received by the controller ofdisplays 208, the controller of displays 208 identifies that display 1belongs to the active zone and displays 2 to p belong to the non-activezone. The controller of displays 208 then proceeds to grey off thenon-active zone. As compared to the non-active zone, the active zone,i.e. display 1, is adjusted or maintained at higher brightness andhigher contrast. Thus, display 1 is operated at an enhanced level ascompared to displays 2 to p because display 1 has higher perceivedbrightness and contrast to the driver as compared to displays 2 to p.

The above scenario is shown by way of illustration of a vehicle cockpitin FIG. 3A and FIG. 3B. Specifically, FIG. 3A shows a prior art displaysystem 200′ wherein all displays (208′-1 to 208′-7) are operated at thesame level of operation, e.g. same brightness, same contrast and in fullcolour. On the other hand, FIG. 3B shows display system 200 according tothe second embodiment of this invention wherein display 208-1 belongs tothe active zone and is maintained at higher perceived brightness andcontrast as compared to displays 208-2 to 208-8 which belong to thenon-active zone and are greyed out.

In a next scenario, driver 210 chooses to gaze again on part of display1 and at the same time moves toward display 1. The controller 206determines and processes the active zone as that part of display 1 (thelocal active zone), while other parts of display 1 are grouped togetherwith displays 2 to p and processed as the non-active zone. Thus, withrespect to display 1, the active zone and non-active zone are within onedisplay. At the same time, movement of driver 210 towards display 1 isdetected by body movement sensor 204 and the movement is processed bycontroller 206 as a gesture command to activate magnification of theactive zone. The controller 206 then transmits the telegrams to thecontroller of displays 208. When the telegram is received by thecontroller of displays 208, the controller of displays 208 identifiesthat the part of display 1 belongs to the active zone while the otherparts of display 1 as well as displays 2 to p belong to the non-activezone. The controller of displays 208 then proceeds to magnify the localactive zone, while keeping the local non-active zone and the globalnon-active zone unchanged (i.e. the other parts of display 1 are notmagnified but still have higher brightness and higher contrast thandisplays 2 to p).

The above scenario is shown by way of illustration of an instrumentcluster display 208-2 in FIG. 3C and FIG. 3D. Specifically, FIG. 3Cshows that display 208-2 is the active zone and is therefore powered on.Driver 210 then gazes at the odometer in display 208-2 and at the sametime moves toward display 208-2. This movement is interpreted as agesture command to activate magnification of the active zone and theodometer is therefore magnified as shown in FIG. 3D.

In yet another scenario, driver 210 chooses to gaze yet again on thepart of display 1 and frowns because he did not obtain the requiredinformation from the magnified local active zone. The frown is detectedby interior camera 202 and processed by controller 206 as a facialgesture indicating a negative reinforcement. The artificial intelligencesoftware compares the incorrectly magnified active zone with the line ofsight sensor data or the gaze position and reassesses the zone to bemagnified. The controller 206 then modifies the area to be magnified andtransmits the telegram to the controller of displays 208. The controllerof displays 208 then proceeds to magnify the updated area for the localactive zone.

FIG. 4 shows an illustration of the architecture of a display system 300according to a third embodiment of this invention. The display system300 comprises p displays 308 (including a navigation display and a headup display); an interior camera 302 and a head movement sensor 303, bothconfigured to determine a driver's line of sight; a controller 306configured to determine an active zone and a non-active zone of the pdisplays 308 based on the driver's line of sight, wherein the p displays308 are configured to operate the active zone at an enhanced level ascompared to the non-active zone. The p displays 308 are connected to anaudio output to allow objects on the display to be accompanied by sound.The controller 306 is further configured to rank content displayed onthe p displays 308 according to an attention score determined by theinterior camera 302 and head movement sensor 303 which are configured tomeasure gaze of driver 310 based on the driver's line of sight. Thecontroller 306 further comprises an artificial intelligence softwareconfigured to use data obtained from interior camera 302, head movementsensor 303 as well as microphone 305 to compare display output withdriver intent.

Here, the navigation display is determined as the active zone, denotedas “a”, and the head up display and displays 3 to p are determined asthe non-active zone, denoted as “n”. Thus, system 300 comprises morethan one display, i.e. p displays, and the active zone comprises onedisplay, i.e. the navigation display.

Say driver 310 is driving to a destination entered into the navigationdisplay and the modification in this case is to suspend the power to thenon-active zone. Driver 310 glances at the navigation display to checkthe route taken, and after the controller 306 determines that the gazeof driver 310 is genuine, the controller 306 determines and processesthe active and non-active zone signal, and transmits a telegram to thecontroller (not shown) of displays 308. When the telegram is received bythe controller of displays 308, the controller of displays 308identifies that the navigation display belongs to the active zone, andthe head up display and displays 3 to p belong to the non-active zone.The controller of displays 308 then proceeds to suspend the power to thenon-active zone, while maintaining the navigation display as a colourdisplay with full brightness and contrast as well as maintainingmovement of the navigation map as the vehicle advances to thedestination. Thus, the navigation display is operated at an enhancedlevel as compared to the head up display and displays 3 to p.

Driver 310 is now driving with his eyes on the road ahead and isfollowing navigation instructions from the navigation display which areoutput via the audio output. Driver 310 is confused by an instructionand tilts his head to the side while looking again at the navigationdisplay. The head tilt and confused facial expression are received ordetected by interior camera 302 and head movement sensor 303 andprocessed by controller 306 as a facial gesture command to activate anincrease in volume with a repeat of the audio navigation instruction.Thus, operation of the active navigation display comprises modifying theaudio output of the navigation display by increasing the volume of theaudio output and repeating the audio output. Further, operation of theaudio output of the active zone is activated upon the facial gesturecommand from the driver.

Interior camera 302 and head movement sensor 303 detects that driver 310not only glances at the navigation display but also his calendar eventsdisplayed on display 3 which displays synced items from his connectedmobile device. The controller 306 keeps count of the number of gazes oneach content (including the calendar events) and the duration of eachgaze on each content to tabulate the attention score of each content.Say the highest ranked content at the end of two minutes is the calendarevents and the head up display was previously selected by driver 310 todisplay the highest ranked content. The controller 306 then directs theaudio output to sound out the question “copy calendar events to head updisplay?” When microphone 305 detects the reply of “yes” from driver310, the controller 306 processes the reply as a voice command toactivate copying and displaying of the calendar events on the head updisplay. Thus, display of the highest ranked content on a selecteddisplay is activated upon the voice command from the driver 310.

1. A display system in a vehicle, the system comprising: one or moredisplays; at least one sensor configured to determine a user's line ofsight; and a controller configured to determine an active zone and anon-active zone of the one or more displays based on the user's line ofsight, wherein the one or more displays are configured to operate theactive zone at an enhanced level as compared to the non-active zone. 2.The system of claim 1, wherein operation of the active and non-activezones comprises modification of an operating parameter of the one ormore displays selected from the group consisting of: illumination;movement of any object on the display; magnification; and combinationsthereof.
 3. The system of claim 2, wherein illumination of thenon-active zone is modified to have at least one of: lower brightness,lower contrast, or lower grey-scale value, as compared to the activezone.
 4. The system of claim 2, wherein the movement of any object onthe display is suspended in the non-active zone.
 5. The system of claim1, wherein: the system comprises one display, the active zone andnon-active zone are within the one display; or the system comprises morethan one display, the active zone comprising one or more displays. 6.The system of claim 5, wherein, where the active zone is within onedisplay, the active zone is magnified.
 7. The system of claim 1, whereinthe one or more displays are connected to an audio output, and whereinwhere an object displayed in the active zone is accompanied by audio,operation of the active zone comprises at least one of: an increasedvolume of the audio output, or a repeat of the audio output.
 8. Thesystem of any preceding claim, further comprising an input means toreceive a command from the user.
 9. The system of claim 8, wherein theinput means comprises: an audio sensor to receive a voice command fromthe user; a selection command on a display; a movement sensor to receivea gesture command from the user; or combinations thereof.
 10. The systemof claim 7, wherein operation of the audio output of the active zone isactivated upon a command from the user.
 11. The system of claim 1,wherein the controller is connected to an artificial intelligencesoftware configured to use data obtained from the at least one sensorand/or the input means to compare display output with user intent. 12.The system of claim 1, wherein the one or more displays is selected fromthe group consisting of: an instrument cluster, a head up display, asecondary display, a digital rear view mirror or one or more digitalside view mirrors.
 13. The system of claim 1, wherein the at least onesensor is selected from the group consisting of: interior camera, headmovement sensor, and body movement sensor.
 14. A controller configuredto rank content displayed on one or more displays in a vehicle accordingto an attention score determined by at least one sensor configured tomeasure gaze of a user based on the user's line of sight.
 15. Thecontroller of claim 14, wherein the attention score is determined by atleast one of: number of gazes on a content or duration of each gaze onthe content.
 16. The controller of claim 14, wherein content is selectedfrom the group consisting of: driving mode, calendar event, routedestination, traffic information, weather, time, and a combinationthereof.
 17. A display system in a vehicle, the system comprising: oneor more displays; at least one sensor configured to determine a user'sline of sight; the controller of claim 14, wherein one of the displaysis selected to display the highest ranked content.
 18. The system ofwherein the controller is configured to rank content displayed on one ormore displays in a vehicle according to an attention score determined byat least one sensor configured to measure gaze of a user based on theuser's line of sight, wherein one of the displays in the active zone isselected to display the highest ranked content.
 19. A method ofoperating one or more displays in a vehicle using the system of claim17, the method comprising: determining, by the at least one sensor, auser's line of sight; determining, by the controller, a ranking ofcontent displayed on each display according to an attention scoredetermined by gaze of the user based on the user's line of sight;selecting one of the displays to display the highest ranked content; anddisplaying the highest ranked content on the selected display.
 20. Amethod of operating one or more displays in a vehicle using the systemclaim 1, the method comprising: determining, by at least one sensor, auser's line of sight; determining, by a controller, an active zone and anon-active zone of the one or more displays based on the user's line ofsight; operating the active zone of the one or more displays at anenhanced level as compared to the non-active zone.
 21. The method ofclaim 20, wherein operating the active and non-active zones comprisesmodifying an operating parameter of the one or more displays selectedfrom the group consisting of: illumination; movement of any object onthe display; magnification; and combinations thereof.
 22. The method ofclaim 20, wherein operating the active and non-active zones comprises atleast one of: decreasing brightness of the non-active zone as comparedto the active zone; decreasing contrast of the non-active zone ascompared to the active zone; decreasing a grey-scale value of thenon-active zone as compared to the active zone; or suspending movementof any object on the display of the non-active zone.
 23. The method ofclaim 20, wherein, where the active zone is within one display,operating the active zone comprises magnifying the active zone.
 24. Themethod of claim 20, wherein, where an object displayed in the activezone is accompanied by audio, operating the active zone comprises atleast one of: increasing the volume of the audio output; or repeatingthe audio output.
 25. The method claim 20, further comprising receiving,by an input means, a command from the user.
 26. The method of claim 25,wherein the input means comprises: an audio sensor to receive a voicecommand from the user; a selection command on a display; a movementsensor to receive a gesture command from the user; or combinationsthereof.
 27. The method of claim 24, wherein when a command from theuser is received, the steps of increasing the volume of the audio outputof the active zone and/or repeating the audio output of the active zoneare activated.
 28. The method of claim 19, further comprising using dataobtained from the at least one sensor and/or the input means in anartificial intelligence software to compare display output with userintent.
 29. The method of claim 20, wherein the one or more displays isselected from the group consisting of: an instrument cluster, a head updisplay, a secondary display, a digital rear view mirror and one or moredigital side view mirrors.
 30. The method of claim 20, wherein the atleast one sensor is selected from the group consisting of: interiorcamera, head movement sensor, and body movement sensor.